AFRL-AFOSR-UK-TR-2017-0005 Multispectral image enhancement through adaptive wavelet fusion

The initial aim of this project was to combine (1) wavelet-based image quality enhancement techniques that have recently been introduced in the literature with (2) our previously developed color remapping technique to enhance the articulation of details in color fused multiband nighttime imagery. After implementing several of the wavelet super-resolution techniques, we observed that their performance was not as good as claimed by the authors. Although processed images sometimes looked sharper, the overall effect was minimal. Moreover, the methods did not work on all images tested. After consulting mathematicians and after extensive discussions with Dr. Pinkus and Mr. Dommett (Wright-Patterson AFB, OH) we decided not to pursue this approach. In consultation with Dr. Pinkus, we identified and decided to investigate the following related research topics: 1. The lack of publicly available registered dynamic multiband imagery forms a serious drawback for the development and optimization of dynamic image fusion algorithms. 2. Guided filters appear promising candidates to replace standard low-pass and bandpass filters in multiscale image fusion schemes. 3. Color grading may give multispectral night vision imagery a more realistic color appearance. Each of these three topics is related to the original research topic, since they all concern the enhancement of details in colored fused imagery for visual inspection. In response to topic 1, we collected a large database of dynamic imagery that was collected previously during several previous field trials with the TNO TRICLOBS (TRI-band Color Low-light OBServation) all-day all-weather surveillance system. We deposited this database in the Figshare open repository (https://figshare.com) and we wrote a paper for an open access journal (PLoS One) describing the dataset. Sharing this dataset with the research community will enable the development and evaluation of (both static and dynamic) image fusion, enhancement and color mapping algorithms. Publishing the dataset in an open access journal will give it wide exposure. In response to topic 2 we developed a multiresolution image fusion scheme based on guided filtering. The scheme combines state-of-the-art performance with a simple implementation and computational efficiency. In addition, we developed a new Alternating Guided Filter, which exactly preserves the edges of large-scale details (no blurring or distortion) while eliminating fine-scale details. Finally, in response to topic 3, we developed an improved color remapping scheme that gives multiband false color images a more realistic natural daytime color appearance. DISTRIBUTION A. Approved for public release: distribution unlimited. 1 Original aim of the project Recently techniques were introduced that claimed to yield “super resolved” up-scaled images by combining standard interpolation techniques with the wavelet transform (Kumar & Nagaraj, 2013, Tripathi & Kirar, 2014). In previous studies we developed a color remapping technique that gives multiband nighttime imagery an intuitive and stable color appearance (Hogervorst & Toet, 2010). We showed that this color remapping procedure enhances scene perception compared to conventional monochromatic nighttime imagery, and may be deployed to drive human fixation behavior (Toet, et al., 2014). The original aim of this project was to combine the abovementioned wavelet-based image quality enhancement techniques with our color remapping technique to further enhance the articulation of details in color fused multiband nighttime imagery. 2 Initial results on wavelet super-resolution We implemented several of the wavelet super-resolution techniques published in the literature (see Kumar & Nagaraj, 2013, Tripathi & Kirar, 2014). However, the results we obtained were mostly disappointing. Although images sometimes looked somewhat sharper, the overall effect was minimal. Moreover, the methods did not work on all images tested. We discussed these initial findings with Dr. Pinkus and Mr. Dommett and we provided them with our code for evaluation. They agreed with our conclusion that the claim that these wavelet based algorithms are capable of enhancing image resolution cannot be substantiated. In addition, we consulted two mathematicians from the Centrum Wiskunde & Informatica in Amsterdam (https://www.cwi.nl), who checked the algorithms and agreed with our findings that these algorithms do not effectively enhance image resolution. In addition, they concluded that there is no mathematical foundational proof that these algorithms should have a positive effect in all general images. Although these algorithms can sometimes make an image look "sharper" they do not increase resolution. As Mr. Dommett noted: “The claims of these types of algorithms contradict the Nyquist Sampling Theorem they imply that you can represent a signal (image) with a lower sampling frequency and then you can generate more information by re-arranging those numbers, applying "some" manipulation, and then re-arranging the numbers back into an image. At best, the new data is a "guess". Without a priori knowledge of the scene, lighting and the image capturing device, the chosen manipulations are no more valid than interpolation. My conclusion is that these algorithms offer no substantial improvement over (for example) bicubic interpolation with post-sharpening.“ Hence, based on these initial findings, and after consulting Dr. Pinkus, we decided not to pursue this line of research any further. After extensive discussions with Dr. Pinkus we decided to continue the project with the following actions: 1. To collect a large set of registered dynamic multiband imagery, recorded during previous field trials with our TRICLOBS multiband camera system, make this set DISTRIBUTION A. Approved for public release: distribution unlimited. available to the scientific community through an open repository, and publish a paper describing this dataset in an open access journal. 2. To integrate recently introduced guided filtering techniques in a multiresolution image fusion scheme. 3. To develop improved color mapping algorithms to give multiband nighttime imagery a more realistic appearance. The results of these investigations will be presented in the next sections. 3 Dynamic Multi-band Image Data Set The fusion and enhancement of multiband nighttime imagery for surveillance and navigation has been the subject of extensive research for over two decades. Despite the ongoing efforts in this area there is still only a small number of static multiband test images available for the development and evaluation of new image fusion and enhancement methods. Moreover, dynamic multiband imagery is also currently lacking. To fill this gap we present the TRICLOBS dynamic multi-band image data set containing sixteen registered visual (0.4 – 0.7μm), near-infrared (NIR, 0.7 – 1.0μm) and long-wave infrared (LWIR, 8 – 14μm) motion sequences. They represent different military and civilian surveillance scenarios registered in three different scenes. Scenes include (military and civilian) people that are stationary, walking or running, or carrying various objects. Vehicles, foliage, and buildings or other man-made structures are also included in the scenes. This data set is primarily intended for the development and evaluation of image fusion, enhancement and color mapping algorithms for short-range surveillance applications. The imagery was collected during several previous field trials with the TRICLOBS (TRIband Color Low-light OBServation) all-day all-weather surveillance system, that was constructed in the course of the previous EOARD-funded project “Registration of a Dynamic Multimodal Target Image Test Set for the Evaluation of Image Fusion Techniques” (Grant FA8655-11-1-3015; see e.g. Dijk et al., 2010; Toet et al., 2010; Toet et al., 2011; Toet & Hogervorst, 2009; Toet & Hogervorst, 2010).This system registers a scene in the Visual, NIR and LWIR part of the electromagnetic spectrum using three optically aligned sensors (two digital image intensifiers and an uncooled long-wave infrared microbolometer). The three sensor signals are mapped to three individual RGB color channels, digitized, and stored as uncompressed RGB (false) color frames. We will make the TRICLOBS data publicly available through the Figshare open repository (https://figshare.com/). Sharing this dataset with the research community will enable the development and evaluation of (both static and dynamic) image fusion, enhancement and color mapping algorithms. To allow the development of realistic color remapping procedures, the data set also contains color photographs of each of the three scenes. The color statistics derived from these photographs can be used to define color mappings that give the multiband imagery a realistic color appearance. Reference c.q. deliverable Toet, A., Hogervorst, M.A., & Pinkus, A.R. (submitted). The TRICLOBS Dynamic Multiband Image Data Set for the Development and Evaluation of Image Fusion Methods. PLoS One. DISTRIBUTION A. Approved for public release: distribution unlimited. 4 Multiscale Image Fusion through Guided Filtering We developed a multiresolution image fusion scheme based on guided filtering. Guided filtering can effectively reduce noise while preserving detail boundaries. When applied in an iterative mode, guided filtering selectively eliminates small scale details while restoring larger scale edges. The proposed multi-scale image fusion scheme achieves spatial consistency by using guided filtering both at the decomposition and at the recombination stage of the multiscale fusion process. First, size-selective iterative guided filtering is applied to decompose the source images into base and detail layers at multiple levels of resolution. Then, frequency-tuned filtering is used to compute saliency maps at successive levels of resolution. Next, at each resolution level binary weighting maps are obtained as the pixelwise maximum of corresponding source saliency maps. Guided filtering of the binary weighting maps with their corresponding source images as guidance images serves to reduce noise and to restore spatial consistency. The final fused image is obtained as the weighted recombination of the individual detail layers and the mean of the lowest resolution base layers. Application to multiband visual (intensified) and thermal infrared imagery demonstrates that the proposed method obtains state-of-the-art performance for the fusion of multispectral night vision images. The method has a simple implementation and is computationally efficient. References c.q. deliverables Toet, A. (2016b). Iterative Guided Image Fusion. PeerJ Computer Science, 2 (e80), 1-26. doi: 10.7717/peerj-cs.80. https://peerj.com/articles/cs-80/ Toet, A., & Hogervorst, M.A. (2016). Multiscale Image Fusion through Guided Filtering. Target and Background Signatures, SPIE-9997. In preparation. 5 Alternating Guided Image Filtering Edge preserving filters aim to simplify the representation of images (e.g., by reducing noise or eliminating irrelevant detail) while preserving their most significant edges. These filters are typically nonlinear and locally smooth the image structure while minimizing both blurring and over-sharpening of visually important edges. We developed the new Alternating Guided Filter (AGF) that achieves edge preserving smoothing by combining two recently introduced filters: the Rolling Guided Filter (RGF) and the Smooth and iteratively Restore Filter (SiR). We showed that the integration of RGF and SiR in an alternating iterative framework results in a new smoothing operator that preserves significant image edges while effectively eliminating small scale details. The AGF combines the large scale edge and local intensity preserving properties of the RGF with the edge restoring properties of the SiR while eliminating the drawbacks of both previous methods (i.e., edge curvature smoothing by RGF and local intensity reduction and restoration of small scale details near large scale edges by SiR). The AGF is simple to implement and efficient, and produces high-quality results. We demonstrated the effectiveness of AGF on a variety of images, and provided a public code to DISTRIBUTION A. Approved for public release: distribution unlimited. facilitate future studies. Applying AGF in a multiresolution framework may lead to improved image fusion schemes. Reference c.q. deliverable Toet, A. (2016a). Alternating Guided Image Filtering. PeerJ Computer Science, 2, e72. doi: 10.7717/peerj-cs.72. https://peerj.com/articles/cs-72/ 6 Coloring multiband night vision images Previously, we presented a method for applying daytime colors to fused nighttime (e.g. intensified and LWIR) imagery (Toet & Hogervorst, 2012). Our color mapping not only imparts a natural daylight appearance to multiband nighttime images but also enhances the contrast and visibility of otherwise obscure detail. As a result this colorizing method leads to increased ease of interpretation, better discrimination and identification of materials, faster reaction times and ultimately improved situational awareness (Toet et al., 2014). A crucial step in this coloring process is the choice of a suitable color mapping scheme. When daytime color images and multiband sensor images of the same scene are available the color mapping can be derived from matching image samples (i.e., by relating color values to sensor signal intensities). When no exact matching reference images are available the color transformation can be derived from the first-order statistical properties of the reference image and the multiband sensor image (Toet, 2003). We therefore investigated various color grading methods to implement this step. We developed an improved color grading scheme leading to colors that better match the natural daytime colors. Reference (c.q. deliverable) Hogervorst, M.A., & Toet, A. (2016). Improved Colour Matching Technique for Fused Nighttime Imagery with Daytime Colours. Electro-Optical and Infrared Systems: Technology and Applications, SPIE-9987: SPIE. In press. DISTRIBUTION A. Approved for public release: distribution unlimited.